Mapping Acoustic and Semantic Dimensions of Auditory Perception

Auditory categorisation is a function of sensory perception which allows humans to generalise across many different sounds present in the environment and classify them into behaviourally relevant categories. These categories cover not only the variance of acoustic properties of the signal but also a wide variety of sound sources. However, it is unclear to what extent the acoustic structure of sound is associated with, and conveys, different facets of semantic category information. Whether people use such data and what drives their decisions when both acoustic and semantic information about the sound is available, also remains unknown. To answer these questions, we used the existing methods broadly practised in linguistics, acoustics and cognitive science, and bridged these domains by delineating their shared space. Firstly, we took a model-free exploratory approach to examine the underlying structure and inherent patterns in our dataset. To this end, we ran principal components, clustering and multidimensional scaling analyses. At the same time, we drew sound labels’ semantic space topography based on corpus-based word embeddings vectors. We then built an LDA model predicting class membership and compared the model-free approach and model predictions with the actual taxonomy. Finally, by conducting a series of web-based behavioural experiments, we investigated whether acoustic and semantic topographies relate to perceptual judgements. This analysis pipeline showed that natural sound categories could be successfully predicted based on the acoustic information alone and that perception of natural sound categories has some acoustic grounding. Results from our studies help to recognise the role of physical sound characteristics and their meaning in the process of sound perception and give an invaluable insight into the mechanisms governing the machine-based and human classifications

Perceptual strategies underlying second language acquisition

The literature suggests that listeners do not pay equal attention to all available acoustic information. Instead, when perceiving speech, they place more importance on some acoustic cues than others (Francis & Nusbaum, 2002). The patterns of weights assigned to different cues appear to change with increased linguistic experience, not only in the first language (L1; Mayo et al., 2003) but also in the second language (L2; Chandrasekaran et al., 2010). However, the role of attention and salience in cue weighting is still under discussion. This thesis presents a series of experiments designed to test this hypothesis in the context of native English speakers and Mandarin Chinese learners of English. First, we compared how prior experience (language background, musical training, and their interaction) shapes cue weighting strategies and tested whether the cue weighting of different cues reflects the direction of attention towards them or their salience. Compared to English speakers, Mandarin speakers showed enhanced attention to and preferential use of pitch across tasks but no increased pitch salience. Effects of musical training were contingent upon participants’ L1. We also demonstrated that perceptual strategies are not consistent across tasks, suggesting they are not driven by domain-general abilities. Second, since acoustic cues play different roles across languages, learning a new language might require listeners to make greater use of L1-irrelevant dimensions. We designed a targeted training focused on redirecting listeners’ attention towards an L2-relevant acoustic cue. Although the observed training effects were not long-lasting, we showed that perceptual strategies in categorizing L2 prosody could be adjusted with as little as three hours of training. This finding has the potential to inform the development of L2 learning paradigms targeting specific auditory challenges experienced by learners. Overall, this thesis provides novel insights into the long-debated role of dimension-selective attention and dimensional salience in cue weighting

Training auditory processing promotes second language speech acquisition

Recent evidence suggests that domain-general auditory processing (sensitivity to the spectro-temporal characteristics of sounds) helps determine individual differences in L2 speech acquisition outcomes. The current study examined the extent to which focused training could enhance auditory processing ability, and whether this had a concomitant impact on L2 vowel proficiency. A total of 98 Japanese learners of English were divided into four groups: (1) Auditory-Only (F2 discrimination training); (2) Phonetic-Only (English [æ] and [ʌ] identification training); (3) Auditory-Phonetic (a combination of auditory and phonetic training); and (4) Control training. The results showed that the Phonetic-Only group improved only their English[æ] and [ʌ] identification, while the Auditory-Only and Auditory-Phonetic groups enhanced both auditory and phonetic skills. The results suggest that a learner’s auditory acuity to key, domain-general acoustic cues (F2 = 1200-1600 Hz) promotes the acquisition of knowledge about speech categories (English [æ] vs. [ʌ])

Automated assessment of second language comprehensibility: Review, training, validation, and generalization studies

Whereas many scholars have emphasized the relative importance of comprehensibility as an ecologically valid goal for L2 speech training, testing, and development, eliciting listeners’ judgments is time-consuming. Following calls for research on more efficient L2 speech rating methods in applied linguistics, and growing attention toward using machine learning on spontaneous unscripted speech in speech engineering, the current study examined the possibility of establishing quick and reliable automated comprehensibility assessments. Orchestrating a set of phonological (maximum posterior probabilities and gaps between L1 and L2 speech), prosodic (pitch and intensity variation), and temporal measures (articulation rate, pause frequency), the regression model significantly predicted how naïve listeners intuitively judged low, mid, high, and nativelike comprehensibility among 100 L1 and L2 speakers’ picture descriptions. The strength of the correlation (r = .823 for machine vs. human ratings) was comparable to naïve listeners’ interrater agreement (r = .760 for humans vs. humans). The findings were successfully replicated when the model was applied to a new dataset of 45 L1 and L2 speakers (r = .827) and tested under a more freely constructed interview task condition (r = .809)

Incidental and Multimodal High Variability Phonetic Training: Potential, Limits, and Future Directions

Scholars have extensively investigated the effectiveness of high variability phonetic training (HVPT), that is, identification and discrimination of second language speech sounds produced by multiple speakers followed by trial-by-trial feedback. Building on the notion of incidental and multimodal learning in cognitive psychology (e.g., Lim & Holt, 2011), we developed a new, HVPT-based videogame paradigm in which participants aimed to shoot clay targets as fast as possible while being guided to learn sound cues as a by-product of planned learning. Focusing on the speech acquisition of 58 Japanese English-as-a-foreign-language learners, the current study set out to test the pedagogical potential and limits of the incidental HVPT approach. According to the results of statistical analyses, the effectiveness of incidental HVPT can be more clearly observed if it focuses on more learnable targets (e.g., acquisition of English [æ]–[ʌ] rather than [r]–[l] contrasts) with gains being more generalizable from trained to new speakers’ voices and from perception to production dimensions

Auditory precision hypothesis-L2: Dimension-specific relationships between auditory processing and second language segmental learning

Growing evidence suggests a broad relationship between individual differences in auditory processing ability and the rate and ultimate attainment of language acquisition throughout the lifespan, including post-pubertal second language (L2) speech learning. However, little is known about how the precision of processing of specific auditory dimensions relates to the acquisition of specific L2 segmental contrasts. In the context of 100 late Japanese-English bilinguals with diverse profiles of classroom and immersion experience, the current study set out to investigate the link between the perception of several auditory dimensions (F3 frequency, F2 frequency, and duration) in non-verbal sounds and English [r]-[l] perception and production proficiency. Whereas participants' biographical factors (the presence/absence of immersion) accounted for a large amount of variance in the success of learning this contrast, the outcomes were also tied to their acuity to the most reliable, new auditory cues (F3 variation) and the less reliable but already-familiar cues (F2 variation). This finding suggests that individuals can vary in terms of how they perceive, utilize, and make the most of information conveyed by specific acoustic dimensions. When perceiving more naturalistic spoken input, where speech contrasts can be distinguished via a combination of numerous cues, some can attain a high-level of L2 speech proficiency by using nativelike and/or non-nativelike strategies in a complementary fashion

Slow phase-locked modulations support selective attention to sound

To make sense of complex soundscapes, listeners must select and attend to task-relevant streams while ignoring uninformative sounds. One possible neural mechanism underlying this process is alignment of endogenous oscillations with the temporal structure of the target sound stream. Such a mechanism has been suggested to mediate attentional modulation of neural phase-locking to the rhythms of attended sounds. However, such modulations are compatible with an alternate framework, where attention acts as a filter that enhances exogenously-driven neural auditory responses. Here we attempted to test several predictions arising from the oscillatory account by playing two tone streams varying across conditions in tone duration and presentation rate; participants attended to one stream or listened passively. Attentional modulation of the evoked waveform was roughly sinusoidal and scaled with rate, while the passive response did not. However, there was only limited evidence for continuation of modulations through the silence between sequences. These results suggest that attentionally-driven changes in phase alignment reflect synchronization of slow endogenous activity with the temporal structure of attended stimuli

Auditory precision hypothesis-L2: dimension-specific relationships between auditory processing and second language learning

Growing evidence suggests a broad relationship between individual differences in auditory processing ability and the rate and ultimate attainment of language acquisition throughout the lifespan, including post-pubertal second language (L2) speech learning. However, little is known about how the precision of processing of specific auditory dimensions relates to the acquisition of specific L2 segmental contrasts. In the context of 100 late Japanese-English bilinguals with diverse profiles of classroom and immersion experience, the current study set out to investigate the link between the perception of several auditory dimensions (F3 frequency, F2 frequency, and duration) in non-verbal sounds and English [r]-[l] perception and production proficiency. Whereas participants' biographical factors (the presence/absence of immersion) accounted for a large amount of variance in the success of learning this contrast, the outcomes were also tied to their acuity to the most reliable, new auditory cues (F3 variation) and the less reliable but already-familiar cues (F2 variation). This finding suggests that individuals can vary in terms of how they perceive, utilize, and make the most of information conveyed by specific acoustic dimensions. When perceiving more naturalistic spoken input, where speech contrasts can be distinguished via a combination of numerous cues, some can attain a high-level of L2 speech proficiency by using nativelike and/or non-nativelike strategies in a complementary fashion

Domain-General Auditory Processing Explains Multiple Dimensions of L2 Acquisition in Adulthood

In this study, we propose a hypothesis that domain-general auditory processing, a perceptual anchor of L1 acquisition, can serve as the foundation of successful post-pubertal L2 learning. This hypothesis was tested with 139 post-pubertal L2 immersion learners by linking individual differences in auditory discrimination across multiple acoustic dimensions to the segmental, prosodic, lexical, and morphosyntactic dimensions of L2 proficiency. Overall, auditory processing was a primary determinant of a range of participants\u27 proficiency scores, even after biographical factors (experience, age) were controlled for. The link between audition and proficiency was especially clear for L2 learners who had passed beyond the initial phase of immersion (length of residence > 1 year). The findings suggest that greater auditory processing skill benefits post-pubertal L2 learners immersed in naturalistic settings for a sufficient period of time by allowing them to better utilize received input, which results in greater language gains and leads to more advanced L2 proficiency in the long run (similar to L1 acquisition).The project was funded by the Kansai University Fund for Supporting Young Scholars 2018.2018年度関西大学若手研究者育成経

Successful second language learning is tied to robust domain-general auditory processing and stable neural representation of sound

There is a great deal of individual variability in outcome in second language learning, the sources of which are still poorly understood. We hypothesized that individual differences in auditory processing may account for some variability in second language learning. We tested this hypothesis by examining psychoacoustic thresholds, auditory-motor temporal integration, and auditory neural encoding in adult native Polish speakers living in the UK. We found that precise English vowel perception and accurate English grammatical judgment were linked to lower psychoacoustic thresholds, better auditory-motor integration, and more consistent frequency-following responses to sound. Psychoacoustic thresholds and neural sound encoding explained independent variance in vowel perception, suggesting that they are dissociable indexes of sound processing. These results suggest that individual differences in second language acquisition success stem at least in part from domain-general difficulties with auditory perception, and that auditory training could help facilitate language learning in some individuals with specific auditory impairments